The Vestibulo Ocular Reflex (herein abbreviated VOR) is a reflex of the human eye whereby head movement causes movement of the eyes in an opposite direction (i.e., to that of the head). As the head moves, the semicircular canals in the ears, which are spatially located in three perpendicular planes, send signals to the brain indicative of the velocity and acceleration of the head in all directions. The brain then sends signals to the muscles of the eye to move in an opposite direction to the direction of head movement. The VOR results in a stabilized image on the retina of the eye as the head moves and allows the eyes to stay aligned and focused on an object even as the head moves.
Some display systems use pixel projection mechanisms for projecting frames comprised of a plurality of pixels on a reflective or semi reflective surface designed to reflect the pixels into the eyes of a user viewing the surface, where not all pixels are projected on the surface simultaneously. The pixels in such display systems can be projected pixel-by-pixel, pixel-row by pixel-row, or in any other manner, in which for at least a given pair of pixels, there is a time difference between projecting the first pixel of the pair and the second pixel of the pair on the surface.
In some cases, the surface on which such display systems project the frames is a see-through display, being a transparent surface, such as a visor of a helmet (or any other head worn system) worn by a user (e.g. a pilot of an aircraft or any other operator of a stationary or moving platform), treated with a semi-reflective treatment making visor a combiner, as is known in the art.
In those cases, where pixel projection mechanisms projects pixels (forming a frame) in a non-simultaneous manner on a see-through display reflecting the pixels onto the eyes of the user, due to the VOR, objects projected by the pixel projection mechanism can appear distorted if the pose of the see-through display on which such objects are projected changes over time during projection of the pixels comprising the objects (while the pixel projection mechanism and the see-through display on which it projects maintain a fixed spatial relationship).
In addition, in some cases, adding to the complexity, it is desirable to display a plurality of elements (e.g. images, videos, text, symbols, or any other type of data that can be displayed on the see-through display) on the see-through display, in such a manner that at least a first element is displayed relative to a first coordinate system while at least a second element is displayed relative to a second coordinate system, other than the first coordinate system. It is to be noted that although reference is made herein to at least two elements, each displayed relative to a corresponding coordinate system, in some cases more than two elements can be displayed relative to more than two corresponding coordinate systems.
In order to ease the understanding of the phenomena, reference in some examples provided herein will be made to a scenario in which certain spatial relationship between two or more elements (comprised within a frame), projected by such display system on the see-through display, and two or more corresponding real-world objects, seen by the user through the see-through display, is required to be maintained. In the scenario, at least a first real-world object is fixed to a first coordinate system (e.g. earth coordinate system), while at least a second real-world object is fixed to a second coordinate system (e.g. a coordinate system of a moving platform). For example, a first element projected on the see-through display may be required to overlay the first real-world object (that is fixed to the first coordinate system) seen through the see-through display while a second element projected on the see-through display may be required to overlay the second real-world object (that is fixed to the second coordinate system) seen through the see-through display. It is to be noted that in other cases, the one or more of elements are not required to overlay the corresponding real-world objects, and instead they are required to be displayed at a certain other spatial relationship therewith (e.g. be parallel thereto). The spatial relationships discussed above is compromised when the pose of the see-through display changes while the pixels are still being projected (i.e. at any given time between projection of the first pixel and the last pixel of a given image on the see-through display).
It is to be noted however that the distortion of the objects projected by the pixel projection mechanism will occur irrespective of any relationship of such objects with real-word objects, in those cases where the pose of the see-through display on which such objects are projected changes over time during the projection thereof.
Attention is drawn in this respect to FIGS. 1A, 1B and 1C. FIG. 1A is an illustration of a desired display of a frame comprising two elements displayed relative to two different coordinate systems, each of the elements overlaying a corresponding real-world object (in the illustrated example—two corresponding vertical lines) seen through the see-through display, in accordance with the prior art. FIG. 1B is an illustration of the rolling display effect occurring upon a change of the see-through display's pose relative to a first coordinate system (e.g. Earth coordinate system (also referred to herein as “real-world coordinate system”, interchangeably)) and, simultaneously, to a second coordinate system (e.g. a coordinate system of a moving platform (also referred to herein as “platform coordinate system”, interchangeably) in which a user wearing the see-through display is located), while a given frame is being projected by the pixel projection mechanism, in accordance with the prior art. FIG. 1C is another illustration of the same rolling display effect, in accordance with the prior art.
Looking at FIG. 1A, a frame 100 is shown, representing the real-world, having real-world coordinate system. A first vertical line 120 is shown within frame 100, being an object within the real-world (e.g. a building). In addition, frame 160 is shown, representing a moving platform in which a user wearing the see-through display is located, the moving platform having its own coordinate system, other than the real-world coordinate system. A second vertical line 140 is shown within frame 160, being an object within the moving platform (e.g. a slider Graphical User Interface (GUI) object shown on a display within the moving platform).
Two elements are projected on the see-through display 110, namely two vertical lines: a first projected vertical line 130 (comprised, in the illustrated example, of four vertical pixels) and a second projected vertical line 150 (comprised, in the illustrated example, of three vertical pixels). The vertical lines, first projected vertical line 130 and second projected vertical line 150, are projected so that: (a) all pixels of the first projected vertical line 130 are perceived by the user viewing the see-through display as vertical, and overlaying the first vertical line 120 within frame 100 representing the real-world, as desired, and (b) all pixels of the second projected vertical line 150 are perceived by the user viewing the see-through display as vertical, and overlaying the second vertical line 140 within the frame 160 representing the moving platform, as desired. However, in the example shown in FIG. 1A, the see-through display's 110 pose does not change during projection of the pixels.
In the illustration shown in FIG. 1B, the see-through display's 110 pose changes over time, as the see-through display 110 moves at a certain angular rate to the left-hand side (changes its spatial position). In addition to the movement of the see-through display 110, the moving platform performs a turn to the opposite side of the movement of the see-through display 110. In the example, the see-through display's 110 movement results in a shift of the see-through display 110, equivalent to one pixel to the left-hand side relative to earth coordinates, and two pixels shift to the left-hand side relative to the moving platform coordinates, during presentation of each pixel row.
More specifically, at T0, being the time the projector projected the first pixel of the first projected vertical line 130 and the first pixel of the second projected vertical line 150 on the see-through display 110 (i.e. the pixel located at the second row, seventh column (denoted in the illustration “first pixel world”), and the pixel projected at the third row, ninth column (denoted in the illustration “first pixel platform”), in the seven by thirteen matrix of pixels projectable by the projector), (a) the first pixel of the first projected vertical line 130 is aligned with the first vertical line 120 (being an object fixed to earth in the illustrated example), and (b) the first pixel of the second projected vertical line 150 is aligned with the second vertical line 140 (being an object fixed to the moving platform in the illustrated example).
However, at T1, being the time the projector projected the second pixel of the first projected vertical line 130 and the second pixel of the second projected vertical line 150 on the see-through display 110 (i.e. the pixel located at the third row, seventh column (denoted in the illustration “second pixel world”), and the pixel projected at the fourth row, ninth column (denoted in the illustration “second pixel platform”), in the seven by thirteen matrix of pixels projectable by the projector), (a) due to the change in the see-through display's 110 pose relative to earth coordinates, the second pixel of the first projected vertical line 130 is no longer aligned with the first vertical line 120, as it is shifted one pixel distance to the left-hand side with respect to the first vertical line 120, and (b) due to the change in the see-through display's 110 pose relative to the moving platform's coordinate system (a change that is, in the illustrated example, different than the change in the see-through display's 110 pose relative to earth coordinates), the second pixel of the second projected vertical line 150 is no longer aligned with the second vertical line 140, as it is shifted two pixels distance to the left-hand side with respect to the second vertical line 140.
At T2, being the time the projector projected the third pixel of the first projected vertical line 130 and the third pixel of the second projected vertical line 150 on the see-through display 110 (i.e. the pixel located at the fourth row, seventh column (denoted in the illustration “third pixel world”), and the pixel projected at the fifth row, ninth column (denoted in the illustration “third pixel platform”) in the seven by thirteen matrix of pixels projectable by the projector), (a) due to the change in the see-through display's 110 pose relative to earth coordinates, the third pixel of the first projected vertical line 130 is no longer aligned with the first vertical line 120, as it is shifted two pixels distance to the left-hand side with respect to the first vertical line 120, and (b) due to the change in the see-through display's 110 pose relative to the moving platform's coordinate system, the third pixel of the second projected vertical line 150 is no longer aligned with the second vertical line 140, as it is shifted four pixels distance to the left-hand side with respect to the second vertical line 140.
At T3, being the time the projector projected the fourth pixel of the first projected vertical line 130 on the see-through display 110 (i.e. the pixel located at the fifth row, seventh column (denoted in the illustration “fourth pixel world”) in the seven by thirteen matrix of pixels projectable by the projector), due to the change in the see-through display's 110 pose, the fourth pixel is no longer aligned with the first vertical line 120, as it is shifted three pixel distance to the left-hand side with respect to the first vertical line 120.
The pixel shifting effect illustrated in FIG. 1B is referred to herein as a rolling display effect, and it results in the projected vertical lines appearing diagonal instead of aligned with the first vertical line 120 and the second vertical line 140. It can be appreciated that due to the fact that the relative motion of the see-through display 110 with respect to earth coordinate system is different than the relative motion of the see-through display 110 with respect to moving platform coordinate system, the slopes of the first vertical line 120 and the second vertical line 140 are different.
It is to be noted that in the example illustrated in FIG. 1B, the see-through display's 110 pose changes at a fixed rate with respect to both earth coordinates and the moving platform's coordinates, however this is not necessarily so, as the rate can be increased or decreased during projection of the pixels of a given frame. The pose change rate also affects the pixel shifting distance, and the higher the pose change rate is relative to the respective coordinate system—the larger the pixel shifting distance is. It is to be further noted that the reference herein to the first vertical line 120 and to the second vertical line 140 is for illustration purposes only, and the rolling display affect is also problematic without referring to real-world objects. For example, an attempt to project any object (e.g. a vertical line), will result in the user, viewing the see-through display 110, perceiving the object in a distorted manner (e.g. a vertical line will appear to the user as diagonal). It is to be further noted that although only two elements are projected on the see-through display 110, in some cases more than two elements can be displayed, while it may be desirable that each will be fixed to a corresponding coordinate system. Furthermore, in some cases, more than two coordinate systems can exist (e.g. moving platform inside another moving platform and earth, etc.)
To further exemplify the rolling display effect problem, attention is drawn to FIG. 1C. In the illustrated example, the see-through display 110 is connected to a helmet worn by a person located within a moving platform (e.g. a pilot of an aircraft). An earth fixed object 15 within the real-world is also shown, fixed to earth coordinates (denoted Ec in the illustration), and a moving platform fixed object 19 is shown, fixed to the moving platform's coordinates (denoted Pc in the illustration). Two elements are projected on the see-through display 110, namely element 11 and element 12. Element 11 is designed to be overlaid on top of the earth fixed object 15, and element 12 is designed to be overlaid on top of the moving platform fixed object 19. Line 17 represents the line of sight of the person located within the moving platform with respect to the earth fixed object 15, and line 20 represents the line of sight of the person located within the moving platform with respect to the moving platform fixed object 19, at T0. T0 is the time the projector projected the first pixel of element 11 and element 12, as can be appreciated looking at the see-through display 110-T0. See-through display 110-T0 shows the projection of the first pixels of element 11 and element 12.
In the illustrated example, the person wearing the helmet rotates his head to the right-hand side, while the moving platform turns to the right hand side, so that at T1, the new line of sight of the person located within the moving platform with respect to the moving platform fixed object 19 (denoted 20A in the figure) is now at a degree a with respect to line 20, whereas the new line of sight of the person located within the moving platform with respect to the earth fixed object 15 (denoted 17A in the figure) is now at a degree 2α with respect to line 17. As a result, the second pixel of element 11 is shifted one pixel to the right, while the first pixel of element 12 is shifted two pixels to the right, as can be appreciated looking at the see-through display 110-T1. See-through display 110-T1 shows the projection of the second pixels of element 11 and element 12. There is thus a need in the art for a new method and system for correcting a rolling display effect.
References considered to be relevant as background to the presently disclosed subject matter are listed below. Acknowledgement of the references herein is not to be inferred as meaning that these are in any way relevant to the patentability of the presently disclosed subject matter.
US Patent Application No. 2016/0189429 (Mallinson), published on Jun. 30, 2016, discloses methods, systems, and computer programs for the presentation of images in a Head-Mounted Display (HMD). One HMD includes a screen, a processor, inertial sensors, a motion tracker module, and a display adjuster module. The motion tracker tracks motion of the HMD based on inertial data from the inertial sensors, and the display adjuster produces modified display data for an image frame to be scanned to the screen if the motion of the HMD is greater than a threshold amount of motion. The display data includes pixel values to be scanned to rows in sequential order, and the modified display data includes adjusted pixel values for pixels in a current pixel row of the image frame to compensate for the distance traveled by the HMD during a time elapsed between scanning a first pixel row of the image frame and scanning the current pixel row of the image frame.
US Patent Application No. 2016/0035139 (Fuchs et al.) published on Feb. 4, 2016, discloses methods, systems, and computer readable media for low latency stabilization for head-worn displays are disclosed. According to one aspect, the subject matter described herein includes a system for low latency stabilization of a head-worn display. The system includes a low latency pose tracker having one or more rolling-shutter cameras that capture a 2D image by exposing each row of a frame at a later point in time than the previous row and that output image data row by row, and a tracking module for receiving image data row by row and using that data to generate a local appearance manifold. The generated manifold is used to track camera movements, which are used to produce a pose estimate.
U.S. Pat. No. 9,595,083 (Smith et al.) published on Mar. 14, 2017, discloses an apparatus for image displaying. The apparatus includes a prediction system and an imaging system. The prediction system is configured to predict, for a first time, a first position of a display device at a specific future time for displaying an image associated with a position of the display, and predict, for a second time that is later than the first time, a second position of the display at the future time with an offset to the first position. The imaging system is configured to render a first image associated with the first position, buffer the first image in a memory, and adjust the buffered first image according to the offset to generate a second image associated with the second position of the display device.
U.S. Pat. No. 5,933,125 (Fernie et al.) published on Aug. 3, 1999, discloses a method for reducing image instability in a virtual environment due to the transport delay of the image generator and other components of the system. A method is given for determining the error in the generated virtual environment and using this error for shifting the image on the display device thus providing a more accurate and more stable representation to the viewer.
U.S. Pat. No. 9,443,355 (Chan et al.) published on Sep. 13, 2016, discloses methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image by circuitry within the display.
PCT Patent Application No. WO/2016/164207 (Crisler et al.) published on Oct. 13, 2016, discloses a method to display video such as computer-rendered animation or other video. The method includes assembling a sequence of video frames featuring a moving object, each video frame including a plurality of subframes sequenced for display according to a schedule. The method also includes determining a vector-valued differential velocity of the moving object relative to a head of an observer of the video. At a time scheduled for display of a first subframe of a given frame, first-subframe image content transformed by a first transform is displayed. At a time scheduled for display of the second subframe of the given frame, second-subframe image content transformed by a second transform is displayed. The first and second transforms are computed based on the vector-valued differential velocity to mitigate artifacts.
US Patent Application No. 2015/0235583 (Schowengerdt et al.) published on Aug. 20, 2015, discloses a user display device comprising a housing frame mountable on the head of the user, a lens mountable on the housing frame and a projection sub system coupled to the housing frame to determine a location of appearance of a display object in a field of view of the user based at least in part on at least one of a detection of a head movement of the user and a prediction of a head movement of the user, and to project the display object to the user based on the determined location of appearance of the display object.